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socially for a successful school experience. Twenty experts in early 
childhood education were invited to attend one of two sessions in a 
university business school team decision productivity laboratory. During each 
session the participants were asked to identify the key characteristics of an 
ideal prekindergarten education program, rate the characteristics of the 
entire group, and identify programs they felt best embodied these 
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effective key programs and produced numerous examples of these programs. The 
power of the decision laboratory was seen in the collaborative work process. 
Using networked computers, participants contributed ideas and opinions on the 
prompts, simultaneously and anonymously, during electronic brainstorming. The 
technology also enabled participants to rate or rank all responses, 
accelerating group consensus while eliminating group intimidation. Appendixes 
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Abstract 

Computerized support for group processes such as analyzing problems and developing 
iimovative solutions is quite common in the business community. However, no evidence can be 
found in the educational literature that these same techniques have been applied to educational 
problems. The purpose of this paper is to present an application of electronic decision-making 
on an educational problem. 

Data suggested that students begiiming kindergarten in a large southern urban area were 
not prepared cognitively or socially for a successful school experience. Twenty experts in early 
childhood education were identified and invited to attend one of two sessions in a university 
business school team decision productivity laboratory. During each session, the participants 

were asked to identify the key characteristics of an ideal pre-kindergarten education program, 

^ , 

rate the characteristics of the entire group, and identify programs they felt best embodied these 
characteristics. The power of the decision laboratory is the collaborative work process. The 
participant-centered computer technology involves everyone in the meeting process. Using 
networked microcomputers, participants contribute ideas and opinions on the prompts, 
simultaneously and anonymously, during electronic brainstorming. The technology also enables 
participants to rate or rank all responses, accelerating group consensus while eliminating group 
intimidation. 

In this example, both groups arrived at similar key characteristics of effective 
pre-kindergarten programs and produced numerous examples of these programs. This 
methodology has a great many potential uses in education. For example, it might be employed 
for focus group analysis, strategic planning, or program evaluation. It has the potential to be an 
effective educational methodology, but needs to be studied in many more situations. 
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Electronic Decision-Making: 

A Potential New Methodology for Educational Research 

Educational researchers have many occasions to determine the consensus of opinion 
among a group of people. Traditionally, they employed techniques such as telephone surveys, 
mail surveys, focus groups, and the Delphi technique. The selection of one of these techniques 
represents a choice between its advantages and disadvantages, often compromising important 
attributes of the study. For example, if individuals' opinions uncorrupted by the input of others 
are important, surveys can be efficient. On the other hand, if sharing the thoughts of others to 
prompt thinking on the issue is important, a Delphi or focus group might be useful. However, 
the Delphi often takes months to complete in the traditional manner and one strongly opinionated 
individual can dominate a focus group. 

Computerized support for group processes such as analyzing problems and developing 
innovative solutions is quite common in the business community. However, little evidence can 
be found in the educational literature that these same techniques have been applied to educational 
problems. The purpose of this paper is to present an application of electronic decision-making 
on an educational problem. We hope this example spurs additional study on this modem 
approach to analyzing problems and developing innovative outcomes. 

Background 

The background for this study is presented in two parts. First, a review of the techniques 
traditionally used by educational researchers to determine consensus of opinions and test new 
ideas is presented. Second, a description of electronic decision-making processes and their 
development is reviewed. 
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Traditional Consensus Estimation Techniques 

A number of techniques are available for determining the level of consensus that exists 
concerning a particular issue. Among the most common techniques are mail or telephone 
surveys, focus groups, Delphi surveys (nominal group technique), and Q-methodology 
(Kerlinger, 1986; Scriven, 1991; Stephenson, 1953). These techniques have been used regularly 
in business and marketing for a variety of purposes including identifying trends, forecasting 
consumer demand, and identifying consumer needs. Each of these strategies has also been used 
for determining levels of consensus concerning issues in education. 

Each of these methods also has characteristic advantages and disadvantages that must be 
considered when determining their worth for a particular consensus estimation study (Kerlinger, 
1986). For example, surveys can elicit feedback about one or more issues from a large, defined, 
and representative sample of a population of interest. This defined sampling can provide the 
researcher with population estimates and confidence intervals for targeted questions. When the 
population of interest is large and the desire is to determine the population parameters for 
variability and central tendency of responses to a set of preselected items and fixed options, 
telephone or mail surveys are an appropriate technique. However, surveys tend to have a 
number of weaknesses that limit their value for consensus estimation. Problems include low 
response rates which increase errors of estimation and decrease the generalizability of findings, 
limitations of one-time “snapshot” sampling, mail surveys can require a month or more in order 
to complete data collection, and the quality of the conclusions (e.g., consensus) is limited by the 
quality of the forced-response items (i.e., the range of items and options provided on the survey 
limit the variety of information that can be solicited). 
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Delphi surveys solicit ratings and recommendations from a limited and specialized group 
of respondents (25-100 respondents is t 5 ^ical). The advantage of the Delphi is that the 
procedure is iterative (Kerlinger, 1986). That is, the typical Delphi procedure involves three 
rounds of data collection and comment from the same set of respondents. The respondents are 
shown the range of responses to an initial set of items as well as any new recommendations. 
Limited comments and justifications are allowed. The process works toward a reduction and 
prioritization of items. The anonymity of the process facilitates consensus-building by avoiding 
some problems associated with interpersonal dynamics (status, territoriality, etc). The greatest 
limitations of the Delphi technique are: (a) the time and costs that are involved in soliciting and 
aggregating multiple rounds of data collection, and (b) that conclusions are constrained by the 
characteristics and representativeness of the sample of respondents (the “expert panel”). 

Focus groups have been a very popular procedure for gathering ideas and opinions about a 
particular issue (Scriven, 1991). Arguably, focus groups are more appropriately viewed as a 
method for identifying issues rather than for estimating consensus. Focus groups involve 
gathering (physically or technologically) a small group of people (6-12) along with a skilled 
facilitator to discuss a set of guiding questions about an issue or object. Focus groups are the 
most dynamic of the methods discussed here in that the group discussion may stray off the 
primary topic of interest, but the facilitator may determine that the information is valuable and 
worth pursuing. Typically, focus groups are comprised of relatively homogeneous sets of people 
with an identified stake in the focus topic (Patton, 1987). More than one focus group is often 
needed in order to either adequately represent key groups of informants or to adequately 
represent any one group of stakeholders. As with any qualitative procedure, one strength of 
focus groups is the richness of the transcriptions generated. 
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Another strength of the procedure is associated with the benefits of “groupthink”. That 
is, ideas build upon one another in an active group process. This often allows new ideas and 
insights to emerge that would not have developed without the interactive process. Just as the 
advantages of focus groups are associated with the qualitative method and group process, so are 
the disadvantages. Focus group findings are constrained by the representativeness, adequacy, 
expressiveness, and richness of the sampled group. Piuposeful sampling within constituent 
populations when forming a specific focus group and across constituent populations when 
determining the range of groups that are needed is important and often problematic. 

Identification of themes from the narrative data is critical to making valid conclusions based on a 
focus group. Such identification requires a skilled evaluator and may be time-consuming. 
Furthermore, without appropriate member checking (confirming with the participants about the 
major issues that emerged from the focus group), the validity of the facilitator’s or evaluator’s 
conclusions may be suspect. Finally, while group processes can facilitate critical and creative 
thinking when the group functions democratically under the guidance of a skilled facilitator, 
group processes can also repress such thinking and meaningful discourse when the group is 
dominated by a small number of individuals. Such domination tends to occur when implicit or 
explicit hierarchies exist amongst group members or when the facilitator does not facilitate the 
discussion effectively. If participants within focus groups do not feel safe and welcome in 
expressing divergent or “risky” opinions or offering atypical ideas or recommendations, the 
group will not contribute new solutions to a problem and any sense of consensus which emerges 
from the group may be a function of a view imposed on the group by a vocal minority. 

Q-methodology involves individuals from a small but representative group of 
respondents (30-80 respondents are typical in Q-methodology studies) sorting a set of statements 
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(on cards) along a bipolar continuum according to a forced distribution (e.g., Gaussian in shape). 
Statements are weighted for each individual. Statistical methods (e.g., factor analysis) are 
applied to the data in order to determine the degree of consensus amongst a group and 
characteristics of individuals who share common rankings of items or item subsets. Some Q- 
methodology studies allow the respondent group (or a parallel group) to generate the list of 
statements that are then to be ordered along a continuum. Others impose a pre-established list of 
statements on the respondents. Proponents of Q-methodology focus on the value of the 
procedure for understanding the subj ective beliefs (realities) of the respondents (Stephenson, 
1953). The method allows the researcher to examine areas of clustering amongst statements and 
respondents. While the data and results of Q-methodology studies are rich and informative 
(helping researchers understand as much about characteristics of respondents that account for 
consensus as characteristics of statements that are clustered), the method suffers from the threats 
associated with; (a) a one-time survey, (b) an a priori set of options (statements), (c) complexities 
of both qualitative and quantitative data analysis, and (d) challenges of sampling of both 
statements and respondents. While the technique has been around for 60 years, it still generates 
debate about central issues such as sampling requirements for items and respondents, analytical 
procedures, and interpretation of results. 

Electronic Consensus and Decision-Making Processes 

In the early 1980s, research into the nature of teams and managerial meetings was 
undertaken at the University of Arizona (Nunamaker, Briggs, Mittleman, & Vogel; 1999). The 
goal of this research was to develop better methods for conducting teamwork and meetings, 
yielding better results and more efficient use of the participants’ time. Research focused on the 




Electronic Decision-Making 8 



development of computer-based tools that would allow participants to conduct their business in a 
more efficient manner, while also leading to better group decision-making and results. 

In pursuing these goals, several generations of software were developed. Early efforts 
focused on the use of minicomputers, but as microcomputers and Local Area Networks were 
perfected, this became the development platform. These developments are embodied in several 
software packages currently available on the market. The premier software package is currently 
Ventana Corporation’s GroupSystems software (Ventana, 1999, October, see Appendix A). 
GroupSystems consists of a suite of electronic mediation tools allowing group activities in 
brainstorming, commenting on ideas, categorizing ideas, evaluating ideas, group writing, and 
multiple criteria evaluation. 

Computer-mediated meetings tend to ameliorate some of the less desirable aspects of 
conventional face-to-face meetings. Anonymity frees participants to express their views without 
regard to the rank or status of the meeting’s participants. Concurrent discussions or comments 
allow everyone to participate and typically require less time than if participants serially express 
their views. A formal voting tool produces a true picture of group consensus, precluding a . 
unilateral decision by the group’s leader. Finally, a record of all the meeting activities is 
available to all of the participants. 

Using the GroupSystems software (Ventana, 1999, October) requires a facility with 
networked microcomputers including a designated server. Each participant has a workstation, as 
well as another workstation assigned to the meeting facilitator or technographer. In addition to 
the investment in equipment, it is also necessary to acquire appropriate meeting-tool software. 
Software such as GroupSystems is usually licensed on a yearly basis, and since it is targeted for 
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corporate use, can be somewhat costly. Thus, investing in this type of facility usually requires a 
substantial financial commitment. 

Research has shown that computer-mediated meetings are usually quite productive. 
Studies at IBM and Boeing have indicated that a 70% increase in meeting efficiency (in dollars 
or time) is not out of reach (Grohowski, McGoff, Vogel, Martz, & Nunamaker, 1990; Post, 1992; 
Vogel, Martz, Nunamaker, Grohowski, & McGoff, 1990). These studies support the use of 
anonymity and concurrent access as valuable tools in a meeting environment. The use of 
computers does not preclude verbal communication; rather it usually results in more productive 
verbal exchanges. 

Computer-mediated meetings have been used to accomplish many things, but most often 
they are used to help develop strategic plans, assist in business process reengineering, canvass 
focus groups, and engage in a variety of decision-making activities in business. In addition to 
these business and administrative uses, there have been a number of efforts to examine the 
technology’s potential as a learning tool in the classroom setting. Some studies have indicated 
that when creatively used for meaningful student issues, there was increased student participation 
and improvement in reading, writing, problem-solving and teamwork skills. It should be noted 
that learning and using the software and facilitation placed added demands and stress on the 
classroom teachers (Sommers, 1999; Walsh, Briggs, Ayoub, Vanderboom, & Glerm, 1996). 

An Application of the Electronic Decision-Making Process 

The performance of beginning kindergarten students in a large southern urban area 
suggested that their pre-kindergarten experience was not at the desired level. Specifically, these 
students were not prepared cognitively or socially for a successful school experience. It became 
apparent that reaching a consensus about the desired elements of pre-kindergarten programs and 
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identifying programs that meet these key elements would be the first step to a solution. The 
assistance of the School of Business faculty was enlisted to take advantage of their electronic 
decision laboratory. 

By checking with School of Education faculty and local school officials, about 15 people 
were identified who worked in pre-kindergarten education or were early childhood learning 
experts. These people were called and invited to attend one of two sessions at the University's 
business school team decision productivity laboratory. During the calls, the process was 
described briefly and each of these people was asked to identify anyone else who they thought 
could contribute. As these new people were called, they were also asked to identify others. This 
continued until the new people who were identified had been identified previously. The final list 
included 30 people, of whom 20 were invited to attend one of two, two-hour sessions that was 
scheduled for the Team Decision Productivity Laboratory. 

The two business faculty members who direct the laboratory assisted in developing a set 
of seven activities for the Laboratory. These items formed the agenda for each session. The 
agenda included: 

• Introductions 

• Identifying Key People 

• Identifying Key Characteristics of Successful Pre-Kindergarten Programs 

• Rating the Importance of the Key Characteristics 

• Identifying Current Programs that Matched These Characteristics 

• Rating the Identified Programs 

Upon arriving at the Laboratory for the exercise, each participant was assigned to his or 
her own computer. The computers were equipped with privacy screens such that only the 
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participant on a particular computer can see his or her own screen. The first task was to have 
everyone introduce themselves and to provide an introduction to the task. As part of this 
introduction, the participants received a brief one-page summary of the broad goals of the 
exercise titled "Ready to Learn: A Goal for All Birmingham Children" (see copy in Appendix 
B). 

The Team Decision Productivity Laboratory included networked computers running 
GroupSystems software (Ventana, 1999, October). The software has great flexibility. An 
individual's response to a prompt may be made available (anonymously) for everyone to view. 
This may jog the memories of others so that they recall more options. The decision laboratory is 
a collaborative work process. The participant-centered computer technology involves everyone 
in the meeting process. Using networked microcomputers, participants contribute ideas and 
opinions on the prompts, simultaneously and "anonymously, during electronic brainstorming. 

The process does not allow for one person, by weight of their assertiveness or status to influence 
other participants unduly. In addition, the process is very efficient since all participants enter 
their thoughts simultaneously. 

Once the participants have input all of the options they can generate, the system allows 
for them to select the most important in one of numerous possible ways. For example, they 
could be asked to pick the "most" important from the list. They could also be asked to rank the 
responses. If there are too many responses to rank, they could be asked to rate the responses 
using a Likert or some other scale. In our example, the participants were asked to list the key 
characteristics of successful pre-kindergarten programs. After listing approximately 100 
characteristics, they were displayed for everyone and duplicates resolved by group consensus. 
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At this point, the participants were asked to rate each characteristic on a 1 to 10 scale where 10 is 
the most important. 

The results were tabulated in a number of ways. The items were sorted in terms of 
importance based on mean responses; a frequency distribution of the top 19 was presented, as 
well as a graph depicting the total number of points each characteristic was assigned. Finally, a 
“top 10” list was arrived at through group consensus (actually, it was a top 1 1 list in this case). 
The list was; 

1. Parent involvement 

2. Developmentally appropriate programming as per NAEYC 

3. Teachers and teacher aides knowledgeable about early childhood education 

4. Warm, safe, and friendly environment 

5. Well-prepared teachers 

6. Continuity between programs from preschool to public school 

7. Low teacher-student ratio 

8. Open 24 hours and weekends 

9. Vision, hearing, and speech screening 

10. Work towards social justice and racial, ethnic, and gender equity 

1 1. Ongoing staff development 

This same process was also used to identify and order current pre-kindergarten programs. The 
process was completed in two sessions with different participants. Final attendance at the 
sessions was 12 and 7. Both groups arrived at similar results. 
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Reactions to the Process 

We will discuss reactions from two perspectives. First, the reactions from the 
participants will be discussed. Second, we will discuss reactions from a methodological 
perspective. 

Participant Reactions 

While no formal data were collected from the participants of these exercises, the 
researchers were present for both sessions and spoke informally to many of the participants 
before, during, and after the sessions. In addition, the two Team Decision Laboratory faculty 
members conducting the session have conducted hundreds such sessions. 

Before beginning, several participants were concerned about using the computers. They 
were not familiar with computers and were, therefore, somewhat intimidated. During the 
orientation, it was explained that all they woifld have to do would be to input their responses on 
the keyboards. They would not have to know how to "operate" the computers. In addition, there 
were four people to help if a problem were encountered. During the sessions, most people had 
no problems whatsoever inputting their ideas and keeping up. Only one person required extra 
help and this was merely pointing out the proper keystrokes for registering a thought or 
navigating through the options. To a person, the participants foimd the process to be a learning 
experience and they appreciated the opportunity to contribute all of their ideas about the topics, 
not having to wait for someone else, or be concerned that someone would find their ideas stupid. 
Everyone who was asked indicated they would be willing to participate in another similar 



session. 
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Methodological Reactions 

From a methodological standpoint, the electronic team decision-making process draws 
from the attributes of surveys, Delphi procedures, and focus groups, but without many of the 
drawbacks. The process has the ability to gain ideas from numerous people, as do surveys, but 
also includes the joint sharing of ideas as does the Delphi technique and focus groups. However, 
the electronic process does not require multiple mailings or gatherings, as does the Delphi or 
allow one strong individual to sway group opinion as in focus groups. 

Using the Team Decision-Making Laboratory not only reduces the time it takes to collect 
the data, it reduces the time to analyze the data. Since much of the data are qualitative, the 
reduction is even more dramatic. In fact, the process uses the knowledge and experience of the 
participants to analyze the data and arrive at consensus during the session. Thus, basic analyses 
are completed before the session is concluded. However, everything that that is input into a 
computer becomes part of the record. This facilitates a reanalysis if that is desired. 

Conclusion 

The use of electronic decision-making software in a laboratory setting has great potential 
for educational researchers. The potential applications of this process go far beyond those 
described in this paper. We can foresee its use in strategic planning for schools and colleges of 
education, public schools and districts, and research projects; in follow-up studies of graduates; 
gaining input from parents and other stakeholders; understanding the goals of an educational 
institution; and many others. We recommend it be used more in education, but we recommend 
that it be studied as it is used. As with anything that is new, the problems and well as the 
benefits need to be examined systematically. 
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GroupSystems® Overview 



GroupSystems is a suite of team-based decision 
software toois with the power and variety to 
heip groups reach decisions. The suite consists 
of the foilowing toois: 




• Standard Tools 



• AOteriniafilve AmaDysSs 



Standard Tools 

The Standard Toois support business needs such as strategic 
pianning, activity based costing, business-process 
reengineering, innovative probiem soiving, product definition, 
knowiedge management, and many more. To support these 
needs, the Standard Toois use group processes such as 
brainstorming, iist buiiding, information gathering, voting, 
organizing, prioritizing, and consensus buiiding. 

You can aiso customize your GroupSystems instaiiation with 
the add-in toois that you need. The foiiowing add-ins can be 
purchased with the GroupSystems Standard Toois or as 
standaione appiications. 



Survey expands the horizons of on-iine surveys. Use Survey 
for face-to-face or distributed groups across iocai area 
networks, e-maii, the Internet, or your company's intranet — 
and then coiiect and analyze resuits with push-button ease. 



AilfteriniaftSve AiraaOysIs 

Alternative Analysis aiiows your group to expiore the strengths 
and weaknesses of strategic pians, seiect candidates, 
determine the impact of a pian on stakehoiders, generate and 
prioritize product requirements — and much, much more. 
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READY TO LEARN: 

A Goal for All Birmingham Children 

Even an infant can perceive every possible sound in any language. By 10 
months, an infant learns to ignore foreign sounds and concentrate on those in his or 
her own language. Two-year-olds can understand simple sentences and follow 
simple instructions. On the average, 90% of the sentences spoken by three-year- 
olds are grammatically correct; many of their errors occur because they apply the 
rules of grammar literally (the English language has as many exceptions as it has 
rules). Many linguists believe that the opportunity for acquiring language begins to 
close by age 6. There is little doubt that children’s learning patterns have been 
established by the ti m e they begin formal education. Thus, we lose many children 
even before we have had the chance to teach them. Therefore, if we are to address 
the problem of at-risk children effectively, we must begin well before they enter 
school as it currently exists. The purpose of this exercise is begin the quest to 
determine what the characteristics of successful pre-kindergarten programs and 
identify some of the programs that are available for the children of Birmingham that 



have those characteristics. 
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